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(54) Animated screen object for annotation and selection of video sequences 



(57) A simple display of items from a list, which 
changes in synchrony with the corresponding video dis- 
play, and permits simple navigation under user control. 
The display includes three control objects positioned be- 
low a video frame window on a display screen. A middle 
displayed object is located in a focus position and cor- 



responds to the video segment cun^ently being played 
in the video window. When play of the segment is com- 
plete, the object located to the right of the focus position 
moves and displaces the middle displayed object, most 
recently displayed within the focus position, to indicate 
a forward progression within the viewed video se- 
quence. 
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Description 

BACKGROUND OF THE INVENTION 



[0001] The present invention relates to computer 
graphics processing, and more specifically to the anno- 
tation and manipulation of video objects by means of a 
graphical user interface object comprising multiple but- 
tons. 

[0002] Graphical user interfaces exist which provide 
elaborate and flexible manipulation of objects and ac- 
tions on computer screens. Such interfaces have been 
developed tp accommodate the power and perfonnance 
available in contemporary processor chips, and to per- 
mit the incorporation of direct manipulation interface 
hardware such as computer mice, trackballs, joysticks 
and the like. 

[0003] One common function required in computer 
systems is the viewing and selection of items from a list. 
The prior art contains a number of examples of user in- 
terface schemes which work well within the convention- 
al computer environment for this task. The most com- 
prehensive of these from the perspective of the cun-ent " 
disclosure Is U.S. Patent No. 5,677,708, which is as- 
signed to Microsoft and is titled "System for Displaying 
a List on a Display Screen," The '708 patent discloses 
the display on a screen, of a list of user-selectable 
items, at least three in number, where the items on the 
edge of the screen are only partially displayed. The user 
can make a selection from one of the items on the 
screen, or scroll the list to move additional items on or 
off the screen. The combination of these actions results 
in a selection, which results in some action by the un- 
derlying user interface software. 
[0004] The patent literature contains other examples 
of movable or animated button selections. For example, 
U.S. Patent No. 5,515,486 which is assigned to IBM and 
is titled "Method, Apparatus and Memory for Directing a 
Computer System to Display a Multi-Axis Rotatable, 
Polyhedral-Shape Panel Container Having Front Pan- 
els for Displaying Objects" describes the graphical pres- 
entation of an apparently three-dimensional container, 
the faces of which represent infomiation or choices to 
the user. U.S. Patent No. 5,940,076. which is assigned 
to Motorola and Is titled "Graphical User Interface for an 
Electronte Device and Method Therefor," describes a 
circular display of choices or features. 
[0005] The need fora complex and highly featured us- 
er interface has moved beyond the computer environ- 
ment, however. Digital technology is moving into the 
field of home entertainment. The new digital television 
standard is encouraging the integration of computer 
technology into home entertainment appliances. One 
example of this convergence is the digital set top box, 
which functions to receive digital signals from cable or 
satellite broadcast sources, and to demodulate, decom- 
press, and convert these signals into analog signals 
viewable on conventional television sets. The new gen- 



eration of these boxes will incorporate disk drives, mak- 
ing them more like computers In their ability to store and 
reproduce digital content. Similarly, computers are be- 
ing used as home entertainment vehicles, incorporating 
5 DVD drives and VCR-like functionality. 

[0006] Consumer television systems present more of 
a challenge for the user interface designer, since the tel- 
evision or the set top box typically has a lower level of 
computer perfonnance, and a paucity of direct manipu- 
^0 lation options. The typical TV remote control utilizes only 
a rocker switch input mechanism or opposing button 
switches for controlling volume or selecting channels, 
rather than the more flexible mouse or track ball inter- 
face. 

15 [0007] Another aspect of digital technology that is en- 
tering the home entertainment maricet is the provision 
of enhanced or interactive television content. Viewers 
are no longer limited to simply viewing a program, nor 
to the simple VCR-like choices of fast fonvard, rewind, 
20 pause and continue. The new interactive television par- 
adigm promises to provide the user with a richer, more 
flexible and engaging experience, through the Incorpo- 
ration of selective navigation through a multiplicity of lin- 
ear and non-Ilnear paths displaying segments of video, 
25 still Images, and other types of entertainment and infor- 
mation content. The selection of paths through video will 
no longer follow the passive start-and-watch model, but 
will permit or require active choices by the viewer. How- 
ever, the typical expected user of interactive television 
30 cannot be expected to learn and master a complex in- 
terface paradigm. Therefore, the interaction control 
must be simple, self-explanatory, intuitive, and direct. 
Furthemiore, the interface must provide a means of as- 
sisting the user In understanding the significance of the 
35 content being viewed, a means of labeling the various 
choices available to the user, assistance in remember- 
ing the path taken through the content, and a mecha- 
nism for visually depicting linear time flow and allowing 
the opportunity to alter the speed and direction of that 
40 flow. 

[0008] None of the existing interface methods com- 
bines the annotated display of multiple relevant choices 
for time-varying content, with a simple means of navi- 
gating the time series. Accordingly the need remains 
^5 for an improved system for user navigation through vid- 
eo content and the like. 



SUMMARY OF THE INVENTION 

50 [0009] The current invention combines a simple dis- 
play of items from a list, which changes in synchrony 
with the con-esponding video display and pennlts sim- 
ple navigation under user control. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
[0010] 

FIG. 1 Is a schematic diagram of video segments to 
be displayed graphed against a timeline. 
FIG. 2 is a schematic diagram of video segments 
physically arrayed on a storage device. 
FIG. 3 Is a schematic flow diagram showing the sev- 
eral branching pathways obtained by selection of 
various video segments using the teachings of the 
invention, 

FIG. 4 is a layout for video display implemented ac- 
cording to a prefen-ed embodiment of the invention. 
FIG. 6 Is the button configuration of FIG. 4 durihg a 
transition sequence. 

FIG. 6 is a screen view with multiple future choices 
Implemented according to an alternate embodiment 
of the invention. 

DETAILED DESCRIPTION 

[0011] The current invention Is adapted for use with 
video display systems, such as those known in the art, 
that include various video segments that are arranged 
to play in some linear (or non-linear) progression to fomi 
a video sequence. An example of such a known video 
display system Is a DVD, or digital video disc, in which 
a user may selectively skip to a desired segment of the 
video movie using the graphic user interface, pro- 
grammed in software on the disc and operable on the 
DVD player to allow user navigation between the seg- 
ments. The segments are delineated by the producer of 
the DVD, where each segment may include one or more 
scenes. A typical DVD movie may have as.many as forty 
designated segments that the user may select and play. 
[001 2] The cu rrent Invention is also operable on other 
video systems such as hard-drive based systems (such 
as those found on some modern set-top boxes) and vid- 
eo-on-demand systems where the video content is con- 
tinuously being downloaded over a network for play In 
response to user Interaction with the network. 
[0013] FIG. 1 illustratesthe organization of a video se- 
quence at 20, comprised of a plurality of video segments 
on timeline 22, here labeled from segments 1 through 
1 1 . In a typical video sequence, say of a movie, playback 
over the video system occurs in a linear fashion by play- 
ing video segment 1 (which corresponds to the first 
scene of the movie) to completion and then Immediately 
playing video segment 2 and so forth until the end of the 
movie is reached after completion of video segment 11 . 
[001 4] While playback of the video sequences 1 -11 in 
order typically occurs by default and without further in- 
teraction by the user with the video display system, it 
may be desirable to provide a video sequence having 
many possible branches and/or endings. In FIG. 1, for 
example, video segment 5 has three different possibili- 
ties 5a, 5b, and Sc. Segment 5a may be, for instance, a 



video segment showing a great battle In the middle of 
the movie video sequence 20. Segment 5b may be, for 
instance, a null video segment of virtually no time that 
essentially skips the battle scene and instead bridges 
5 segment 4 with segment 6. This may be useful If, for 
instance, one of the viewers is a young child who is sen- 
sitive to violence. Finally, segment 5c may show an ex- 
tended and unedited version of video segment 5a that 
may have initially been cut from the movie released in 

10 the theaters. 

[0015] FIG. 1 also shows segment 1 1 with three pos- 
sible endings 11a, lib, and 11c to the video sequence 
20. As will be appreciated below, only one of these end- 
ings will be selected by user interaction with the video 

IS display system using the selection and annotation 
scheme disclosed in the present invention and dis- 
cussed in more detail further below. 
[0016] FIG. 2 illustrates the physical arrangement of 
data for the video segments on a storage medium such 

20 as a digital video disc (DVD) or hand disk drive. As it is 
desired for the read head of the storage medium to 
traverse the least distance, video segments are typically 
arranged sequentially in the expected order in which 
they are to be played. In the present example, the video 

25 sequence (movie) fomned by the default organization of 
the video segments - that is, without further user inter- 
action as described further below - results in segment 
5a (the battle scene) being played immediately after 
segment 4 and before segment 6. Similarly, the default 

30 ending 11 a to the video sequence is stored immediately 
after the data for segment 10 since the expectation is 
that a viewer of the video segment would chose to view 
the default ending. The other endings, segments 11b 
and 11c, are stored after default ending segment 11a. 

35 [001 7] FIG . 3 illustrates the an-angement of video seg- 
ments into one of a plurality of possible video sequenc- 
es, where the sequence shown is produced by combin- 
ing a total of five video segments . The sequence branch - 
ing follows a series of predefined "rules" that regulate 

40 the flow of the sequence from beginning to end. For in- 
stance, in the example shown, all possible video se- 
quences begin with a display of video segment 1 . From 
there, the sequence has three possible branch paths: to 
segment 2a along the double-lined an^ow, to segment 

45 2b along dashed arrow, and to segment 2c along single- 
lined an-ow. The simplified user interface for selecting 
2a, 2b, or 2c is explained further below. If segment 2a 
is selected (or if 2a Is the default segment absent further 
user interaction), then segment 2a Is displayed on the 

50 video display system - such as an attached television or 
computer monitor and the like - and the sequence pro- 
ceeds to segments 3a, 4a and then 5a before ending. 
Note that the rules associated with selection of segment 
2a require that no further branching exists once the path 

55 designated by the double-lined arrows is chosen. 
[001 8] If segment 2b is selected, as by using the user 
interface of the present invention, then play proceeds to 
2b and thence to segment 3b. If segment 2c is selected, 
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then play proceeds to 2c and thence to segment 3b. The 
rules for selection of segment 2b or 2c disallow segment 
3a from appearing in the rightmost position on the dis- 
play screen as a possible branching option. At 3b, the 
user of the video display system is again given a choice 
of branching paths. 

[0019] The system is programmed to defaultto one of 
the paths in the absence of further user Interaction with 
the video display system. In one example, a user who 
proceeded down the viewing path denoted by the 
dashed line would default to segment 4b after viewing 
segment 3b because, for instance, scenes within video 
segment 4b may better correlate with scenes shown In 
earlier-viewed segment 2b. A user who proceeded 
down the viewing path denoted by the single-lined arrow 
would default to segment 4c for similar reasons, it is also 
possible that both paths could default to the same seg- 
ment (e.g. segment 4b). Finally, segment 4c branches 
into three different paths - segment 5c, segment 5d, and 
segment 5e - with one of the segments being set to de- 
fault and the remaining ones selected under control of 
user input. 

[0020] The current Invention is intended for the anno- 
tation and interactive display of video segments. Typi- 
cally such segments are viewed as they are played, and 
the annotated labels are displayed in sequence, in syn- 
chrony with tfie video display. However, the invention 
may also be used In a mode where the video is not being 
played (having been paused, for example, or prior to in- 
itiating playbacic). In this case, the displayed images will 
correspond to still images taken from the video seg- 
ments corresponding to the annotations. 
[0021] To display the contents of a list of annotations, 
the current invention uses at most three buttons or but- 
ton-sets (refen-ed to herein as "control objects") dis- ^ 
played on a video display screen 24 as shown in FIG. 
4. The video screen has displayed thereon a video seg- 
ment within a video frame 26 and, in a preferred embod- 
iment, a plurality of control objects 28, 30 located in non- 
overiapping relation to the video frame 26 in which the 4 
video segment is displayed. The center object 28 is lo- 
cated within a centrally located focus position 32, des- 
ignated by dashed line, which represents the current fo- 
cus of the list and corresponds to the video selected by 
the user and/or displayed on the screen. 4 
[0022] Several aspects of this diagram are remarl<a- 
bie. First, at most three buttons or button sets are dis- 
played on the screen, each of them shown in its entirety. 
Only in special cases, such as that in FIG. 4, are less 
than three buttons shown: Forthe first segment of video, si 
the 'previous' (left) button is not shown. Similarly, forthe 
last segment of video, the 'next' (right) button is not 
shown. In the rare case that the video contains only one 
segment, only one button would be shown, 
[0023] Second, the center button, which represents ss 
the video currently showing, or last shown prior to 
pause, is highlighted to indicate its selection. The se- 
lected button is always displayed in the center position. 



If the user chooses, in the display shown in RG. 5, to 
select either the 'previous' (In this case, segment 1) or 
'next' selection (here, segment 3), and does so by 
means of selection arrows era rocker switch on atypical 
5 remote control, thefocus shifts to the corresponding but- 
ton, and the button itself moves to the center position. 
This movement maybe accomplished by animating the 
movement, using a multiplicity of intemiediate positions, 
or by simply repainting the labels of the buttons, replac- 
10 ing the label of the center button with the label from the 
selected button, and updating the corresponding labels 
for the 'previous' and 'next* buttons. 
[0024] FIG. 5 shows how a transition state may ap- 
pearfollowlng the selection of the 'next' button when the 
15 button movement is animated. The selection and anno- 
tation method practiced according to a prefen-ed em- 
bodiment of the invention displays a first control object 
28, associated with a displayed first video segment 1 of 
the video sequence, on the display screen 24 in the fo- 
?o cus position 32 simultaneous with the display of the first 
video segment 1 on the display screen within the video 
frame 26. A second control object 30, associated with a 
second video segment, is displayed on the display 
screen 24 adjacent to the focus position. Here, since the 
?5 video segment 2 associated with second control object 
30 would normally be shown within a video sequence at 
a later time than the video segment 1 associated with 
first control object 28, the second object 30 is shown to 
the right of object 28. It is understood that the invention 
0 is not limited to the an-angement of the objects and that 
any arrangement of objects can be implemented so long 
as the arrangement is temporally consistent. For In- 
stance, English language speakers read from left-to-left 
and therefore it is more appropriate to have the flow of 
5 time in the same direction. Speakers of Hebrew (right- 
to-left) and of Japanese (up-to-down) may necessitate 
a different organization to make It more intuitive to the 
natural directional flow of the language. 
[0025] During a transition between video segments, 
' the second control object 30 moves to the focus position 
32 (centered and directly below video frame 26), and 
the first control object 28 moves out of the focus position 
toward the left in the direction of arrow 33, in substantial 
synchronicity with a transition between the display of the 
first video segment 1 and the second video segment 2 
on the display screen 24. IVtovement is denoted by the 
repeating dashed portions to the right of each object. 
Following the transition, the right button (control object 
34) appears with the label appropriate for associated 
video segment 3. 

[0026] The first, second, and third objects (28, 30, and 
34, respectively) correspond to current, future and past 
segments respectfully within the default video se- 
quence. It is significant to note that the f onward transition 
(that is, to segments later in time) can be done automat- 
ically as the video sequence is played. These transi- 
tions, as well as backward transitions, may also be 
made under user control. Transitions can occur when 
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video is paused, tliat is, when a still image is displayed. 
[0027] The focus position 32 is interposed between 
the now-moved first control button 28 and the third con- 
trol button 34. Once video segment 2 has completed 
play, the following sequence of events occur associated s 
control object 30 is moved to the left out of the focus 
position 32, leftmost first object 28 is moved off the dis- 
play screen 24, third control object 34 moves into the 
focus position, and a fourth object (not shown in FIG, 5) 
is simultaneously moved Into the position previously va- 
cated by the third control object 24. The object move- 
ment occurs in substantial synchroniclty with the start of 
the next video segment 3 and the end object position 
results in the focus position being interposed between 
the control object 30 and the fourth (not shown) control ^5 
object. 

[0028] The control objects are signified by annota- 
tions, such as still images and/or descriptive text, cor- 
responding to the content of the video segments asso- 
ciated with the control objects. The object buttons may 20 
be labeled in a variety of ways, including images, Icons, 
animations, l<eywords, game clues, trivia answers. Such 
labels need not specifically define or describe the cor- 
responding video segment, but will be related to it in 
some way. FIG. 1 includes examples of such annota- 25 
tions, where the object associated with segment 5a is 
annotated "Battle", 5b as "Sl<ip Battle", 5c as "Extended 
Battle", 11a as "Happy Ending", lib as "Sad Ending", 
and 11c as "Extended Happy Ending". 
. [0029] Turning next to FIG. 6, shown Is the video dis- 30 
play 24 (television, computer monitor, cell phone, etc.) 
having displayed thereon control objects 36, 38 and 40 
associated with video segments 9, 10 and 11 respec- 
tively. Control object 40 Includes a plurality of subobjects 
(40a, 40b, and 40c) corresponding to video segments 3S 
11a, 11b and 11c, each of which provide a selectable 
branching pathway from the previous video segment, 
designated by control object 38. Objects 40a, 40b and 
40c correspond to video segments of alternate endings 
to the video sequence/movie. In the absence of Input 
from a user of the video display system and at the end 
of video segment 10, a preset one of the three subob- 
jects (e.g. object 40b) moves to the focus position 32 
and the con-esponding video segment to that object 
(segment 1 1 b) plays within the video frame 26. ^5 
[0030] User selection of one subobjects (or of another 
of the objects) occurs by scrolling among the plurality of 
control objects based on input from a user of the video 
display system, as by using the rocker switch on a re- 
mote control. The designated object is selected as by so 
depressing an 'enter* or some similar function key on the 
remote control and the selection of one of the control 
objects (or subobjects) is accepted by the video display 
system. The selected object Is then moved to the focus 
position and the associated video segment is played 55 
within the video frame window 26. Scrolling is accom- 
plished using visual cues such as a focus frame 42, 
shown highlighting a particular control object 11c in FIG. 



1 , which is moveable between the plurality of objects to 
provide a visual indication of user control of the selected 

control object. 

[0031] The selection process may also be accom- 
plished by means of a screen object, such as a pointer 
or Indicator, which Is manipulated by any standard user 
Interface pointer device like a mouse or trackball; by 
means of designated keys, such as the '4' and '6' keys 
on a telephone touch pad; or directly by means of a 
touch-sensitive display screen. 
[0032] As shown In FIG. 6, the number of subobject 
buttons may be increased, with two or more buttons on 
each side of the center (focus) button. Navigation would 
then be a two-dimensional process, with various options 
available for past (left) and/or future (right) movement. 
As described above, segment lib would be the seg- 
ment displayed if no user intervention occurred, while 
segments 11a and 11c would be optional choices avail- 
able to the user. In this scenario, multiple branching 
paths through the video segments would be possible. 
Navigation in such an interface might require a joystick 
or othercontrol capable of selecting from muitiple direc- 
tions, rather than a simple right arrow or rocker switch 
control, or could be accomplished with the use of rapid 
multiple button presses, or combinations of simultane- 
ous button presses, or alternative keys such as the '3' 
and '9' keys on a touch pad, Page Up/Page Down, or 
others. 

[0033] The buttons might be animated with an indica- 
tion of the progression of time within the particular seg- 
ment named by the button. This may take the fonn of a 
line or bar with a mark or coloration change that moves 
along or across the graphic to mark the relative position 
within the segment. Buttons may provide additional 
functionality. Including pop-up menus for additional in- 
fomnation or selections. 

[0034] Having described and illustrated the principles 
of the invention in a preferred embodiment thereof, it 
should be apparent that the invention can be modified 
in arrangement and detail without departing from such 
principles. We claim ail modifications and variation com- 
ing within the spirit and scope of the following claims. 



Claims 

1 . In connection with a video display system capable 
of displaying a sequence of video segments, a 
method for displaying a plurality of control objects 
associated with said video segments on a display 
screen comprising the steps of: 

displaying a first control object, associated with 
the displayed first video segment, on the dis- 
play screen in a focus position simultaneous 
with the display of a first video segment on the 
display screen; 

displaying a second control object, associated 
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with a second video segment, adjacent to tlie 
focus position; and 

nr^oving the second control object to the focus 
position, and the first control object out of the 
focus position, in substantial synchronlcity with 
a transition between the display of the first vid- 
eo segment and the second video segment on 
the display screen. 

2. The method of claim 1 , further comprising the step 
of displaying a third control object adjacent to the 
focus position, whereby the focus position is inter- 
posed between the second control object and the 
third control object. 

3. The method of claim 1 , further comprising the steps 
of: 

scrolling among the plurality of control objects 
based on input from a user of the video display 

system; 

accepting the selection of one of the plurality of 
control objects based on input from a user of 
the video display system; and 
displaying a video segment associated with the 
selected control object. 

4. The method of claim 1 , further comprising display- 
ing a focus frame within said focus position, said 
focus frame operative to supply a visual indication 30 
of user control of the first control object and moving 
said focus frame under user control to the second 
control object. 

5. The method of claim 1 , said first and second control 35 
objects including displayed therein visual annota- 
tion corresponding to the content of the video seg- 
ments associated with said control objects. 

6. The method of claim 2 wherein the first, second, and 40 
third objects con-espond to curent, future, and past 
segments respectively within a default video se- 
quence. 

7. The method of claim 6, further comprising the step 45 
of simultaneously moving the third object off of the 
display screen, and a fourth object onto the display 
screen simultaneous with the movement of the first 
object out of the focus position and the second ob- 
ject into the focus position so that the end position so 
results in the focus position being interposed be- 
tween the first control object and the fourth control 
object 

8. The method of claim 1 , further comprising the step ss 
of displaying at the second control object a plurality 

of subobjects, each corresponding to a respective 
video segment, to provide a selectable branching 



pathway from the video segment associated with 
the first control object. 

9. The method of claim 8, further comprising the step 
of, in the absence of input from a user of the video 
display system, moving the preselected one of the 
second control subobjects to the focus position, and 
the first control object out of the focus position, at 
the end of the display of the first video segment. 



20 



25 



6 



EP 1 239 672 A2 



5c 



EXTEffDED 
•BATTLE 



fTsKIpTI 5b 

[Uattle/P' 




42 

— — 4~— 

EXTEtfDED 
HAPPY ENDING.: 



VIDEO 
SEGMENTS 



22 



TIME 



FIG. 1 



BATTLE 
(DEFAULT) 



1 I 2 M ) 4 ) 5a ) 6 



FIG. 2 




SAD 
ENDING 

< I 

LP :| Ila I Ub I lie 



f t 

HAPPY EXTENDED 

ENDING HAPPY ENDING 
(DEFAULT) 




2a 



3a 



4a 



2c 







4b 








3b 




4c 





Sa 



5c 



5d 



5e 



2 3 4 5 

VIDEO SEGMENT # ^ 
IN SEQUENCE FIG. 3 



EP 1239 672 A2 



24 



VE)EO 
FRAME 



T 

26 



28- 



r- 



■-I 32 



SEGMENT 


1 > 


SEtrMENT 


#1 


1 
1 
1 





30 



FIG. 4 



VIDEO 
FRAME 



24 

u 



33 



26 



SEGMENT 
#1 


— I — \. 

1 :r 
1 % . 

L— 1 


SEGMENT 


-» i; 
1 1 
J J 

— 1 I- 


SEGMENT 

: n 


• r. 

c 
1 






^36 




34^ 





FIG. 5 




FIG. 6 



